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ABSTRACT 

Ths paper describes the results of a questionnaire 
sent out to 495 institutions of higher education to determine current 
evaluation practices in innovative colleges and universities. Of the 
495 questionnaires mailed, 375 were returned. The results of this 
survey indicate that a majority of the chosen institutions now have 
soae form of mechanism and that this mechanii^m is being used in 
institutional decision making. However, it is not at all clear 
whether the political clout of the office doing the evaluation is 
such that people listen to the recommendations of the evaluation 
specialist. The fact that the data is being used in decision making 
does not necessarily mean that the decisions being made are good 
ones. It was also noted that the sample of institutions, carefully 
selected on a number of criteria that would lead one to believe that 
they were innovative, have still not moved as consistently as 
expected toward a rigorous and analytical evaluation scheme. (HJH) 
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i «>r several vcMis, wc at tfie Center (lave been inter- 
•.'steil III the rapid mciease i»t iiiteiest tn the area of evalu- 
atK)n nt iiistitutiitns ot higher education. Siune ot this in- 
crease is ilue to nev^ pre>Mires tor accountability, coming 
fitun le>:islatuies, parents, trustees, and the student who 
^ ''consumes'* hi>ihei education and wants his her nu>iiey's 
J WMtih. But in addition. M>me institulu>ns seemed l<) be gen- 
J umelv interested in developing better evahiation procedures 
as an end in itself, as a wav ol unpri>ving their performance 
as insnrurions. We were inrerested muhat institutions were 
douik: m terms mI evaluar!«»n practices, v^hat use was being 
made »»f rfie ..-suits, and what kinds ot problems v^ere being 
encountered 

I his papet desc ribes the results ot a questionnaire 
sent luit h\ the Center Uw Research and I)eveli>pmerit in 
lliizhef l iltuaiitMi 'Ui Inne ^. P>^4. to 4^^5 institutions ot 
higher evhication in tfie ^tninir>.* Rather than sending 
questionnaitos r^^ all institutions m the nation, we 

Inniied onr sampit* i«» those insfuuiions that we identified 
as dtMUM siMneihin^: dittereni and perhaps "iimovative" in 
hieliei oiluvaiion at the undergraduate level. Our sources tor 
instiruiions were the lists i>r inemheiship provided by the 
StKietv tor liu\nv.ition m Higher |-ducation. lists ot 
gr.ints rriMii tl)e \ urui tt»r the hnpriAement ot Postsecondary 
fducation, jful hsi< .»f grants awarded bv various toun- 
datit'ns. as well .in msiituiitwis kru^wn to rnetiibeis ot the 
Ntaf i as d«>tng ir\novaMve pri^graintiung at the undergiaduate 
level 

Out teeinij: was that :t anvone was doing interesting 
auii .umtuanf ev.jluath'n. boili .it uuhvidual student grtmth 
aiul .»t pffCf.tms wirluf! an msriiuiit»rK this^collectum ot 
inst!iuth»ns w.Mild prohahlv he the best i!ri>up ti> survev. 
IhuN "Ut n!sTituMt»ns wore w^X nati»Miallv repiesentative 

' I hi\ pr>M<Ht v\ ix tiifi.tf*^t >n I jif tnr ff'tn |h\* Ot*p trttni'iit <»t 
Hi« iti!i \ *\\%\ «n.it« it« l vVWf.irf <>th<*f {Hihli« itiiMtv fr«»tn Ihis pf»»|ft? 
Iff lt%tr\l -m $Mrf»*% 1 ifi.! H. 
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but were systematically biased toward those institutions 
that had demonstrated some commitment to innovative ap- 
proaches in undergraduate education, on the assumption 
that they would also be iimovative about their evaluation 
pro''edures. 

or the 495 questionnaires mailed, we eventually re- 
ceived 375. for a return rate of 75,8%. However, 33 of 
these questionnaires were returned too late to be included 
in the analysis, leaving us with 342 institutions (699^) as the 
basis for the analysis. We did ask that the institution iden- 
tify itself by control and level of offerings. Control and 
level of offerings are presented here simply to give the read- 
er an idea <)f what the response group looked like in terms 
of those two dimensions. Six of the 342 respondents did 
not identify their institutions in this way, and thus some of 
the tabulations are based on 336 cases. 



TABLE A 

Responding Institutions by Control and Highest Degree 



Control 


No. 


Percent 


Public 


164 


48.8 


Private 


172 


51.2 


Level of Offerings 






Ooctorate 


83 


24.7 


Beyond MA. !ess than doctorate 
MA 




25.3 


4-5 years BA 


93 


27.7 


2 years 


75 


22.3 



In terms ol the jK^rsons who tilled out question* 

runes. 101 were trom mstitutu^nal research i>tYices, w3 from 

the a^^adennc dean's otYice, 50 Irom the president's iWt'ice. 

42 trom a vuc president's oftlce (usuallv academic), and 4" 

lh>m a miscellanv administrative otfices. 
) 

U0 



\ 



i ummitment tn tvaluatUm Resi^an h 

i»t' the niajoi questions wc used to get at this 
vaiidbte was whether or not the institution had an insti- 
tutional research otTive with a tull-time director. We found 
that 175 of our instituUims (about 51%) did have an insti- 
tutional research otTice with a full-time director, white se* 
ven indicated that they had an IR office with a part«time 
diiector. Thus, only about 5,V^ of our institutions, even 
though they were selected for tiielr innovative character* 
htu's. had an office which was devoted to the systematic 
evaluation of programs, (This is our somewhat idealized 
view of the role of institutional research offices.) Institu- 
tional research offices in our admittedly biased sample are 
nu)re prevalent in the public sector (105 of 164) than they 
are in the private (68 of 172). 

Another interesting result of our study is that ai* 
though 5.Vy i)f the responding institutions indicated that 
they have an institutional research office with either a full* 
or part*tiine director, only 30% of the respondents to *he 
survey indicated that they were IR directors. The question- 
naire (sent to the president's office) apparently was routed 
to Si>me individuals who felt that they should respond ra* 
ther than the director of institutional research. What this 
tells one about the political role of the directors of institu- 
tional research offices on campus can only be speculated 
about, but one wonders if some of them are perhaps seen as 
clerical functionaries rather than line administrators. 

We were also interested in knowing, as an index of 
commitment to evaluation on campus, what percent of the 
total operating budget of the institution was devoted to 
evaluation. Here we got the largest '*no response" item in 
imr questionnaire. Two hundred and eighty-one of our 342 
institutions indicated that they could not respond to the 
item. This suggests ( 1 ) the amounts spent on evaluation 
were ttu) small to mention, or (2) most institutions cannot 
break out expenses for evaluation as requested. We suspect 
that (2) IS nu»st common. We wanted to know whether or 
not an institution-wide committee on evaluation was part 
of the campus committee structure. One hundred and six of 
i>ur institutions (317f) had such a committee, while 236 
(b^J'ri said that no institution-wide committee on 'evalu- 
ation existed. 

iuaUn l\i'uiuution 

The primary mode for faculty evaluation of courses 
was for the individual faculty member Xomitute the evalu- 
ation process in his class. (St>me, of course, use students to 
collect and analyze the data ) Two hundred and fifty-six 
institutions indicated that evaluation efforts initiated by 
individual faculty were their mam pattern (75% of our sam- 
pled whereas for 25'^ no faculty efforts at course e\alu- 
jtion were described. However, 225 institutions (66%) had 
s*>?ne fi»rm of student evaluation of faculty done by cam- 
PNs wiJe questionnaires, usually 'liotne grown" rather than 
produced nationally. One hundred and fiftynme of our in- 
stitutions (44'^) have more than individually developed fac- 
iiltv evaluation instruments, but less than institution-wide, 
usudllv done b> simie departments but not all. An obvious 
pr4>blern lot the institution which allows the varit>us depart- 



ments to develop their own evaluation designs concerns the 
difficulty of developing some sort of compatible framework 
so that evaluation standards can be maintained across the 
institution. 

Evaluation Procedures Used in 
'Innovative " Programs 

We also asked our respondents to describe the dif- 
ferent evaluation procedures that were used for innovative 
programs. The response rate here was only 77 institutions; 
140 answered that they did have different evaluation proce- 
dures for innovative endeavors but failed to describe them. 
Of the 77, 10 named programs themselves (such as 4-M or 
competency); 32 indicated speciflc techmques that were 
used (such as portfolios or interviewing); 6 named evalu- 
ation designs; and 29 named evaluation stmctures, such as a 
faculty committee. Here, a higher percentage of reitponses 
came from private insihuttnns than was true of our sample 
as a whole, indicating perhaps that private institutions are a 
little further along in terms of being specific about evalu- 
ation designs for innovative programs. 

Use of Standardized Research Instruments 

We asked whether any standardized instruments had 
been used for campus assessment since 1970, or whether 
instruments were developed locally; the results were inter- 
esting and are presented in Table B, In each of the cate* 



TABLE B 
Instrumtiits Used 

%of 



Studsnt Charactsriitics 


No. 


342 


OPt 


48 


14.0 


MMPI 


34 


10.0 


Other instruments 


St 


14.9 


Locally Oeveioped 


78 


22.8 




211 


61.7 


Environmental Measures 






CCI 


19 


5.6 


CCA 


4 


1.2 


ACE (CiRP) 


86 


25.1 


CUES 


52 


15.2 


HEMEK 


18 


5.3 


Other 


14 


4.1 


Locally Oeveioped 


88 


17.5 




167 


48.8 


Institutional Goals/Functioning 






IGI 


69 


17.3 


IF! 


45 


1J.2 


Other 


13 


3.8 


Locally Oeveioped 


43 


12.6 




IbO 


46.8 


Course Evaluation 






SIR 


18 


5.3 


ISS 


17 


5.0 


Eval. Univ. Teaching 


4 


1.2 


CEO 


8 


2.3 


Other 


13 


3.8 


Locally Oeveioped 


193 


56.4 




253 


74.0 



$i)fm of inHtrunients us«({ Un measuring student character- 
isu^x characteristics of the campus environment, institu* 
luma) goaii and tunctioiiing, and course evaluations, institu* 
turns clearly prefer kK;ally developed instruments to those 
that are nationally available. Tliis is particularly true in the 
area oi student evaluations ot faculty performance in 
courses. Although there are problems with commercially 
produced instruments, one wonders at the efHcacy of devel- 
oping •*home grown" instruments in each of these cate- 
gories. Many of the locally developed faculty evaluation 
forms, foi example, are coniposed of highly ambiguous 
Items and very poor scoring procedures (for example, if 
only 5'^ oi*the class fills out the questionnaire, the data is 
often presented with resptmUenr penentages only, with no 
real numbers, allowing no one to know how many of the 
total class actually responded \o the questionnaire). 

I or certain kinds of evaluations, a good locally de< 
veloped instrument can be suptMior, but for assessment of 
environmental characteristics, kHrally develo|)ed instru- 
ments may be doin^ as much harm as they are doing good, 
depending on the skill of the individuals making them up. 
One Ci>nclusnm from our data is thai much remains to be 
done in finding out more about the quality of the locally 
develt>ped instruments. Those that we have seen leave much 
to be desired, and the advantages of local instruments have 
yet to be established. Ht)wever, it is quite clear that locally 
devehiped instruments are still very widespread in each of 
these tour areas. It may also be that many institutions do 
x^K^\ have gin^d access to the range of good instruments al- 
ready in existence. This is easily remedied.^ 

The Use ipf Evaiuation Data 

We were 4'specialiy interested in knowing how the 
variety of data p;athcred by campuses and presented in Ta- 
ble B ^vasbem^nsed in institutional decision-making. These 
are iun]< uv^Ien*! below m Table C, with the actual 



TABLE C 
list of AsMument Results 

(n « 142 inttitutioiu) ^ ^ 

%of 

Utt of Assessment Rtmitt No^ 342^ 

55.8 
40.9 
33.3 

29.2 
26.6 
22.2 
5.6 



5.3 

3.8 

3.2 
2.9 
2.6 
.9 



*sec i^.g.. H«HlKliin%t>n. J. Hurtt. H. I eviiie. and S. Brinl. A 
%l4tnud/ tH^ t vahtanori of Innovativt f>oKranti and Prai Hces m 
Htj(hrr f'timatttm. iVntrr f«>r Kinearih and Development in Migher 
Kluv4tH9n. rni%^fMt> of i altfornu. Berkeley, l*>74. 
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number of institutions reporting the use of each one, plus 
the percentage ot 342, the total respondent group for this 
item. 

One of the difficulties with this table is that we 
cannot be $ure exactly what data was used for each pur- 
pose. More than likely student evaiuation questionnaire 
data was used for faculty promotion and tenure decisions, 
while the environmental assessment measures were pro* 
bably used in decisions about the development of residen- 
tial units. Some of the incoming freshmen personality data 
were probably used in new curriculum and counseling pro- 
grams. However, we cannot be sure of this based on the data 
we have. Table C does suggest, however, that the data 
gathered from evaluation efforts of this sort have been used 
rather widely across a number of institutional decision* 
making area^. We ran the data also by control of institu- 
tions and by level of highest offerings to see if there were 
any significant differences in the rank order of how assess* 
ment results were used, and found that there were no signi- 
ficant differences* 

Current Evaluation Problem Areas 

We were particularly interested in knowing where 
our respondents felt more evaluation work needed to be 
done. Presented below in Table D are the current mnjor 
problem areas in which respondents were currently work* 
ing. The best interpretation of this data is to assume that 
these are evaluation problems that the institution has not 
yet successfully dealt with; in other words, institutions are 
attempting to find a satisfactory evaluation strategy for 
dealinii^ with these particular problems. For the most part, 
the responses probably indicate a lack of satisfaction with 
existing evaluation strategies and instruments in these areas. 



TABLE 0 
Evaiuation Probltm Amis 



Evaluation Problem Areas 


No. 


342 


Evaluating effectlveneu of new programs 


249 


72.8 


Developing cost effectiveness measures 


197 


57.8 


Studying the learning environment 


185 


54.1 


Using results in decision-making 
Studying institutional goals 


178 


52.0 


168 


49.1 


Translating life experience 


113 


33.0 


Seif-studies for accreditation 


113 


33.0 


Studying management structure 


109 


31.9 


Interpreting assessment results 


93 


28.7 



Clearly, the most important evaluation problem our 
respondents must deal with is that of evaluating the etfec* 
tiveness of new or modified programs. Two hundred and 
forty-nine indicate that this is a current difficulty. Fol- 
lowing close behind is the need to develop cost-cflective- 
ncss measures. 

Studying the institution's learning environment md 
goals IS important in 1X5 ot* our campuses, as are the pro- 
blems of using evaluation results m campus decision- 
making. However, only 17H out ot 342 indicated that lie 
latter is a problem, which suggests that in over halt the 
campuses successful use of evaluation data is now being 

4 3 



institutional tong-r^nge planning 191 

Faculty promotion and xnmrn 140 

Budget devilopment 114 
Offvdop nnw cufncuium/counsalnii 

programs 100 

Reallocation of program funds 91 

New admissions procedures 76 

Develop new residential units 19 

Ot her Specified U»s 

New programs/Curncutum development 18 
Faculty developmtnt/improvement of 

instruct ofi 13 
Ott>6r iniututionai matten (e.g.* short- 
range planning, IR pnonties) 1 1 
Smdent piacement/services/act^vities 10 
Acc redt tart on/Sel I study 9 
AHocatton of resources 3 



uccDinplhhcd in terms ot the campus decision-making pro* 
ve&&. (Or, Mo attempt has been made to ui^e such data ) 

Ihe piohierm ot credit tor lile experience involved 
Humewhat fewer ut ()ur iriHtitutions. Whether thb is because 
the> have satisfactory ways of deahng with these pr(^blems 
or whether it is because the institutions have not yet had to 
deal with the grarMing of credit tor hfe experience cannot 
he determined from ovr data, although our dehbera:ely 
biased sample would suggest the former. 

It IS also interesting that institutions indicated 
that interpreting the results of assessment data was a fairly 
maior problem. One suspects, on the basis of no evidence, 
that one of the interpretation problems concerns the fact 
that /their instrumentation may have been *'home grown.** 
In this case, because there are no norms with which their 
per/ormance tan be compared, it is highly likely that dif- 
ticuhies would be discovered in terms of figuring out what 
the results meant for the institution. Another possible inter- 
pretation, oi course, is that people who arc responsible for 
evaluation efforts have not received proper training and do 
not know how to interpret the data given. This is specu- 
lation, however, as we have no evidence that would indicate 
the precise reason for the difficulty in interpreting results. 

Areas in Whu h \ew Measures Were Needed 

We also asked institutions what kinds of new mea- 
sures would be ot greatest assistance to them, and in what 
areas. Hero, our u\s itutions were not quite so responsive. 
We received 270 prop.)sed areas of measurement from the 
342 respondin]|i institutions. There was very little consensus 
within the group in terms of what kinds of measurements 
needed to be developed. However, they could be classified 
mto tour or tive major headings, including effectiveness 
measures (7^)). new rniHiets (11). student selection (2 IK 
ne\^ performance measures of teachers and staff ( 144). and 
measures nt the envirtnunenl (15). 

This data can be taken as some small appeal from 
practitioners lor better evaluation instruments in these 
areas. Certamlv, the authors i)f this article agree that theie 
are some o\ ihc crucial areas in which new evaluation in* 
Ntruments and strategies have t*) be developed. However. 
rhiN does t.ut mean that rtationally standardized question- 
ruiieN are tlie only wa> to prt>ceed. Indeed, for several of 
thov areas Ipersonal growth and teaching competency), the 
•Marulaid" apprt»ach ma> be the least useful, and some 
tlu)uv!hr could be p\c\\ to other locally normed strategres 
such as gaming and stmuilation. Q-sorts. self-anchoring 
scales. un4>btrusue measures, pr(»|ective techniques, group 
evaluations, etc. 

The results ot this survey indicate that a majority of 
oui mstttuiions now have some torm of evaluation mech- 
antsui .uut that this mechanism is being used in institutu)nal 
deusion-makiu^. However, it is not at all clear whether the 
political '\lotM" ot the lattice doin^ the evaluatum is such 
that pei»ple tisrcn to the recommendations ot the evaluation 
Npe^.ul:st \hc ta^.t that the data is being iscd in decision- 
makinif does wot necessarily mean that decisions made 
are gtMHl t»nes. It i» also somewhat surprising that our 
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sample of institutions, carefully selected on a lumber of 
cnteria that would lead one to believe that the wereS(jin- 
novative/* have still not moved as consistently as we had 
expected toward a rigorous and analytical evaluation 
scheme. If these are experimental colleges, the evaluation 
design pn^^edures certainly do not seem to match their 
interest in curriculum innovation, and without successful 
checking out of the impact jf curricular change, how will 
an institution know whether the change was worthwhile or 
not? 
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